Kubernetes Configuration Management: A Deep Dive
category: Kubernetes Certification
tags: cka, kubernetes, exam, kubectl, certification
The Fundamental Problem: Why Configuration Management Matters
Before diving into Kubernetes-specific solutions, let's understand the core problem. Applications need configuration data to function - database URLs, API keys, feature flags, environment-specific settings. The challenge is making this data available to containers while maintaining security, flexibility, and operational sanity.
The Anti-Pattern: Hardcoding configuration directly into container images creates brittle, insecure, and unmaintainable applications. You'd need different images for dev/staging/prod, secrets would be baked into images, and any config change would require rebuilding and redeploying.
ConfigMaps: The Foundation of Non-Sensitive Configuration
What ConfigMaps Really Are
ConfigMaps are Kubernetes objects that store non-confidential data in key-value pairs. Think of them as a dictionary that lives in your cluster, accessible to any Pod that needs it.
Why Use ConfigMaps Instead of Environment Variables?
The Problem with Hardcoded Environment Variables:
# DON'T DO THIS - Hardcoded in deployment
spec:
containers:
- name: app
env:
- name: DATABASE_HOST
value: "prod-db.company.com" # Now you need different manifests per environment
- name: LOG_LEVEL
value: "INFO"
The ConfigMap Solution:
# ConfigMap - Centralized configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: production
data:
database.host: "prod-db.company.com"
database.port: "5432"
log.level: "INFO"
feature.newUI: "true"
app.properties: |
# Multi-line configuration file
server.port=8080
spring.datasource.url=jdbc:postgresql://prod-db.company.com:5432/myapp
logging.level.com.company=INFO
Deep Example: Real-World ConfigMap Usage
Let's say you're running a microservice that needs different configurations across environments:
# Development ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: payment-service-config
namespace: development
data:
# Simple key-value pairs
PAYMENT_GATEWAY_URL: "https://sandbox.stripe.com"
MAX_RETRY_ATTEMPTS: "3"
TIMEOUT_SECONDS: "30"
DEBUG_MODE: "true"
# Complex configuration file
application.yml: |
server:
port: 8080
payment:
gateway:
url: https://sandbox.stripe.com
timeout: 30s
retry:
maxAttempts: 3
backoff: exponential
logging:
level:
com.company.payment: DEBUG
# JSON configuration
features.json: |
{
"enableNewCheckout": true,
"enableApplePay": false,
"maxTransactionAmount": 1000
}
---
# Production ConfigMap (same structure, different values)
apiVersion: v1
kind: ConfigMap
metadata:
name: payment-service-config
namespace: production
data:
PAYMENT_GATEWAY_URL: "https://api.stripe.com"
MAX_RETRY_ATTEMPTS: "5"
TIMEOUT_SECONDS: "10"
DEBUG_MODE: "false"
application.yml: |
server:
port: 8080
payment:
gateway:
url: https://api.stripe.com
timeout: 10s
retry:
maxAttempts: 5
backoff: exponential
logging:
level:
com.company.payment: WARN
Why This Approach Works:
- Environment Parity: Same application code, different configs per environment
- Centralized Management: All config in one place per environment
- Version Control: ConfigMaps can be stored in Git and applied via CI/CD
- Hot Reloading: Some applications can reload config without restart
Secrets: Handling Sensitive Information
The Critical Difference: Why Secrets Exist
Secrets solve the fundamental security problem of storing sensitive data. While ConfigMaps are stored in plain text in etcd, Secrets provide:
- Base64 encoding (not encryption, but obfuscation)
- Memory-only storage in Pods (not written to disk)
- RBAC integration for access control
- Audit trail capabilities
Deep Example: Database Credentials Management
# Creating a Secret for database credentials
apiVersion: v1
kind: Secret
metadata:
name: database-credentials
namespace: production
type: Opaque
data:
# Base64 encoded values
username: cG9zdGdyZXNfYWRtaW4= # postgres_admin
password: c3VwZXJfc2VjdXJlX3Bhc3N3b3JkXzEyMw== # super_secure_password_123
connection-string: cG9zdGdyZXNxbDovL3Bvc3RncmVzX2FkbWluOnN1cGVyX3NlY3VyZV9wYXNzd29yZF8xMjNAcHJvZC1kYi5jb21wYW55LmNvbTo1NDMyL215YXBw
---
# TLS Secret for HTTPS
apiVersion: v1
kind: Secret
metadata:
name: tls-secret
namespace: production
type: kubernetes.io/tls
data:
tls.crt: LS0tLS1CRUdJTi... # Base64 encoded certificate
tls.key: LS0tLS1CRUdJTi... # Base64 encoded private key
Why Use Different Secret Types?
Kubernetes provides several Secret types for different use cases:
# Docker registry credentials
apiVersion: v1
kind: Secret
metadata:
name: registry-secret
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5jb21wYW55LmNvbSI6eyJ1c2VybmFtZSI6ImRlcGxveWVyIiwicGFzc3dvcmQiOiJzZWNyZXQiLCJhdXRoIjoiWkdWd2JHOTVaWEk2YzJWamNtVjAifX19
---
# Service account token
apiVersion: v1
kind: Secret
metadata:
name: service-account-token
type: kubernetes.io/service-account-token
Why This Matters:
- Kubernetes can automatically inject registry secrets into Pod specifications
- TLS secrets integrate with Ingress controllers
- Service account tokens enable secure pod-to-pod communication
Configuration Injection Patterns: Environment Variables vs Volume Mounts
Environment Variables: When and Why
Environment variables are perfect for simple configuration values that applications read at startup.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
spec:
template:
spec:
containers:
- name: app
image: company/payment-service:v1.2.3
env:
# Direct value injection
- name: SERVICE_PORT
value: "8080"
# From ConfigMap - individual keys
- name: PAYMENT_GATEWAY_URL
valueFrom:
configMapKeyRef:
name: payment-service-config
key: PAYMENT_GATEWAY_URL
# From Secret - individual keys
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: database-credentials
key: password
# Bulk injection from ConfigMap
envFrom:
- configMapRef:
name: payment-service-config
# Bulk injection from Secret
- secretRef:
name: database-credentials
When to Use Environment Variables:
- Simple key-value configuration
- Application reads config once at startup
- Legacy applications expecting environment variables
- 12-factor app methodology compliance
Limitations of Environment Variables:
- Limited to simple strings
- Visible in process lists (
ps aux) - No hot reloading without pod restart
- Size limitations (typically ~32KB total)
Volume Mounts: For Complex Configuration
Volume mounts are ideal for configuration files, certificates, and scenarios requiring hot reloading.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-proxy
spec:
template:
spec:
containers:
- name: nginx
image: nginx:1.21
volumeMounts:
# ConfigMap as volume - entire config
- name: nginx-config
mountPath: /etc/nginx/conf.d
readOnly: true
# ConfigMap as volume - specific file
- name: app-properties
mountPath: /app/config/application.properties
subPath: application.properties
readOnly: true
# Secret as volume - TLS certificates
- name: tls-certs
mountPath: /etc/ssl/certs
readOnly: true
# Secret as volume - specific file with custom name
- name: database-config
mountPath: /app/secrets/db.conf
subPath: connection-string
readOnly: true
volumes:
# ConfigMap volumes
- name: nginx-config
configMap:
name: nginx-configuration
# Set file permissions
defaultMode: 0644
- name: app-properties
configMap:
name: app-config
items:
- key: application.yml
path: application.properties
mode: 0600
# Secret volumes
- name: tls-certs
secret:
secretName: tls-secret
defaultMode: 0400
- name: database-config
secret:
secretName: database-credentials
items:
- key: connection-string
path: db.conf
mode: 0600
Real-World Example: Multi-Tier Application Configuration
Let's configure a complete web application stack:
# Frontend ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: frontend-config
data:
nginx.conf: |
server {
listen 80;
server_name app.company.com;
location / {
root /usr/share/nginx/html;
try_files $uri $uri/ /index.html;
}
location /api {
proxy_pass http://backend-service:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
app-config.js: |
window.APP_CONFIG = {
API_BASE_URL: '/api',
FEATURE_FLAGS: {
ENABLE_DARK_MODE: true,
ENABLE_NOTIFICATIONS: true
},
ANALYTICS_ID: 'GA-XXXX-XXXX'
};
---
# Backend ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: backend-config
data:
DATABASE_HOST: "postgres-service"
DATABASE_PORT: "5432"
DATABASE_NAME: "myapp"
REDIS_HOST: "redis-service"
REDIS_PORT: "6379"
LOG_LEVEL: "INFO"
application.yml: |
server:
port: 8080
spring:
datasource:
url: jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}
username: ${DATABASE_USER}
password: ${DATABASE_PASSWORD}
redis:
host: ${REDIS_HOST}
port: ${REDIS_PORT}
logging:
level:
com.company: ${LOG_LEVEL}
---
# Database Credentials Secret
apiVersion: v1
kind: Secret
metadata:
name: database-secret
type: Opaque
data:
username: bXlhcHBfdXNlcg== # myapp_user
password: cGFzc3dvcmQxMjM= # password123
---
# Frontend Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 3
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: nginx
image: nginx:1.21
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d/default.conf
subPath: nginx.conf
- name: app-config
mountPath: /usr/share/nginx/html/config.js
subPath: app-config.js
volumes:
- name: nginx-config
configMap:
name: frontend-config
- name: app-config
configMap:
name: frontend-config
---
# Backend Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
spec:
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: app
image: company/backend:v2.1.0
ports:
- containerPort: 8080
env:
# From ConfigMap
- name: DATABASE_HOST
valueFrom:
configMapKeyRef:
name: backend-config
key: DATABASE_HOST
- name: DATABASE_PORT
valueFrom:
configMapKeyRef:
name: backend-config
key: DATABASE_PORT
- name: DATABASE_NAME
valueFrom:
configMapKeyRef:
name: backend-config
key: DATABASE_NAME
# From Secret
- name: DATABASE_USER
valueFrom:
secretKeyRef:
name: database-secret
key: username
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: database-secret
key: password
volumeMounts:
- name: app-config
mountPath: /app/config/application.yml
subPath: application.yml
volumes:
- name: app-config
configMap:
name: backend-config
Advanced Configuration Patterns
1. Configuration Hot Reloading
Some applications can reload configuration without restart. Here's how to enable it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: configurable-app
spec:
template:
spec:
containers:
- name: app
image: company/configurable-app:latest
volumeMounts:
- name: config
mountPath: /app/config
# Enable config file watching
env:
- name: CONFIG_WATCH_ENABLED
value: "true"
- name: CONFIG_RELOAD_INTERVAL
value: "30s"
volumes:
- name: config
configMap:
name: app-config
Why This Matters:
- Zero-downtime configuration updates
- Faster iteration during development
- Reduced deployment complexity for config-only changes
2. Configuration Templating with Init Containers
For complex configuration generation:
apiVersion: apps/v1
kind: Deployment
metadata:
name: templated-app
spec:
template:
spec:
initContainers:
- name: config-generator
image: company/config-templater:latest
env:
- name: ENVIRONMENT
value: "production"
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: config-template
mountPath: /templates
- name: generated-config
mountPath: /output
command:
- /bin/sh
- -c
- |
# Generate configuration from templates
envsubst < /templates/app.conf.template > /output/app.conf
containers:
- name: app
image: company/app:latest
volumeMounts:
- name: generated-config
mountPath: /app/config
volumes:
- name: config-template
configMap:
name: config-templates
- name: generated-config
emptyDir: {}
3. Secret Management Integration
For production environments, integrate with external secret management:
# Using External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.company.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "myapp"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: database-secret
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: database/prod
property: username
- secretKey: password
remoteRef:
key: database/prod
property: password
Security Best Practices: The "Why" Behind Each Decision
1. Principle of Least Privilege
# RBAC for ConfigMap/Secret access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: config-reader
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["app-config", "shared-config"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["app-secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: app-config-binding
subjects:
- kind: ServiceAccount
name: app-service-account
roleRef:
kind: Role
name: config-reader
apiGroup: rbac.authorization.k8s.io
Why This Matters:
- Applications can only access the specific configs they need
- Prevents lateral movement in case of compromise
- Audit trail for config access
2. Secret Rotation Strategy
# Deployment with secret rotation awareness
apiVersion: apps/v1
kind: Deployment
metadata:
name: rotation-aware-app
spec:
template:
metadata:
annotations:
# Force pod restart when secret changes
secret-hash: "{{ .Values.secretHash }}"
spec:
containers:
- name: app
image: company/app:latest
env:
- name: SECRET_RELOAD_SIGNAL
value: "SIGUSR1"
volumeMounts:
- name: secrets
mountPath: /app/secrets
# Health check that validates secret validity
livenessProbe:
exec:
command:
- /app/health-check
- --validate-secrets
initialDelaySeconds: 30
periodSeconds: 60
volumes:
- name: secrets
secret:
secretName: app-secrets
3. Configuration Validation
# ConfigMap with validation
apiVersion: v1
kind: ConfigMap
metadata:
name: validated-config
annotations:
config.kubernetes.io/validation: |
type: object
properties:
database:
type: object
properties:
host:
type: string
pattern: "^[a-zA-Z0-9.-]+$"
port:
type: integer
minimum: 1
maximum: 65535
required: ["host", "port"]
required: ["database"]
data:
config.yaml: |
database:
host: "postgres.company.com"
port: 5432
features:
enableNewUI: true
maxConnections: 100
Operational Considerations: Making It Work in Practice
Configuration Drift Prevention
# GitOps approach - all configs in version control
# Directory structure:
configs/
├── base/
│ ├── configmap.yaml
│ └── secret-template.yaml
├── environments/
│ ├── development/
│ │ ├── kustomization.yaml
│ │ └── config-patch.yaml
│ ├── staging/
│ │ ├── kustomization.yaml
│ │ └── config-patch.yaml
│ └── production/
│ ├── kustomization.yaml
│ └── config-patch.yaml
Monitoring and Alerting
# ServiceMonitor for configuration-related metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: config-metrics
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
path: /metrics
interval: 30s
Key Metrics to Monitor:
- Configuration reload frequency
- Failed configuration validations
- Secret rotation events
- Configuration drift detection
Troubleshooting Configuration Issues
# Debug configuration in running pods
kubectl exec -it pod-name -- env | grep -E "(DATABASE|API)"
kubectl exec -it pod-name -- cat /app/config/application.yml
kubectl exec -it pod-name -- ls -la /app/secrets/
# Check ConfigMap/Secret contents
kubectl get configmap app-config -o yaml
kubectl get secret app-secrets -o jsonpath='{.data}' | base64 -d
# Verify volume mounts
kubectl describe pod pod-name | grep -A 10 "Mounts:"
Decision Framework: When to Use What
Use ConfigMaps When:
- Non-sensitive configuration data
- Environment-specific settings
- Application properties files
- Feature flags
- Static content (HTML, JS, CSS)
Use Secrets When:
- Database passwords
- API keys
- TLS certificates
- SSH keys
- OAuth tokens
- Any credential or sensitive data
Use Environment Variables When:
- Simple key-value pairs
- 12-factor app compliance
- Legacy application compatibility
- Container startup configuration
Use Volume Mounts When:
- Complex configuration files
- Binary data
- Hot reloading requirements
- File-based configuration (nginx.conf, etc.)
- Multiple configuration files
Use External Secret Management When:
- Enterprise security requirements
- Centralized secret governance
- Automatic secret rotation
- Compliance requirements (SOX, PCI-DSS)
- Multi-cluster secret sharing
Common Pitfalls and How to Avoid Them
1. Configuration Sprawl
Problem: Too many ConfigMaps and Secrets making management difficult. Solution: Use consistent naming conventions and namespace organization.
2. Secret Leakage
Problem: Secrets exposed in logs or environment variable lists. Solution: Use volume mounts for secrets, implement proper logging practices.
3. Configuration Drift
Problem: Manual changes causing inconsistency between environments. Solution: GitOps workflows, immutable infrastructure principles.
4. Performance Issues
Problem: Large ConfigMaps causing slow pod startup. Solution: Split large configs, use init containers for processing.
5. Update Propagation
Problem: Configuration changes not reflected in running applications. Solution: Implement proper restart strategies, use deployment annotations.
The key to successful Kubernetes configuration management is understanding not just the "how" but the "why" behind each approach. Choose the right tool for each use case, implement proper security practices, and maintain operational visibility into your configuration state.
Last updated: 2025-08-26 20:00 UTC